Liquid cooling system for stackable modules in energy-efficient computing systems

ABSTRACT

A computing system is provided. In the computing system, a plurality of modules physically arranged in a three dimensional hexadron configuration. In the computing system, the at least one module is either a liquid-tight module filled with a non-conductive liquid coolant or a module cooled with a liquid coolant circulating through cold plates mounted on electronic components. In the computing system, the liquid coolant is circulated in a closed loop by at least one pump through a plurality of hoses through at least one of a plurality of heat exchangers. In the computing system, the plurality of heat exchangers is coupled to an exterior portion of the surface of the computing system. In the computing system, the plurality of heat exchangers cool the liquid coolant through tinned tubes exposed to the surrounding air.

This application is a divisional of application Ser. No. 12/788,863,filed May 27, 2010, status allowed.

BACKGROUND

The present application relates generally to an improved data processingapparatus and method and more specifically to mechanisms for a liquidcooling system for stackable modules in an energy-efficient computingsystem.

High-performance computing (HPC) uses supercomputers and computerclusters to solve advanced computation problems. The HPC term is mostcommonly associated with computing used for scientific research. Arelated term, high-performance technical computing (HPTC), generallyrefers to the engineering applications of cluster-based computing (suchas computational fluid dynamics and the building and testing of virtualprototypes). Recently, HPC has come to be applied to business uses ofcluster-based supercomputers, such as data intensive, commercialanalytics applications, and transaction processing.

However, many HPC systems are hindered by limits in the powerconsumption, space, cooling, and adaptability. That is HPC systems arecomposed out of thousands of components which occupy considerable space,require considerable cooling, use massive power, and are not readablydeployable.

SUMMARY

In one illustrative embodiment, a computing system is provided. In theillustrative embodiment, the computing system comprises a plurality ofmodules physically arranged in a three dimensional hexadronconfiguration. In the illustrative embodiment, the computing system alsocomprises at least one module of the plurality of modules is at leastone of a liquid-tight module filled with a non-conductive liquid coolantused to cool the at least one module or a module cooled with a liquidcoolant circulating through cold plates mounted on electronic componentswithout requiring the module to be filled by the coolant material. Inthe illustrative embodiment, the liquid coolant is circulated in aclosed loop by at least one pump through a plurality of hoses through atleast one of a plurality of heat exchangers. In the illustrativeembodiment, the plurality of heat exchangers is coupled to an exteriorportion of the surface of the computing system. In the illustrativeembodiment, the plurality of heat exchangers cool the liquid coolantthrough finned tubes exposed to the surrounding air.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exampleembodiments of the present invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The invention, as well as a preferred mode of use and further objectivesand advantages thereof, will best be understood by reference to thefollowing detailed description of illustrative embodiments when read inconjunction with the accompanying drawings, wherein:

FIG. 1 depicts a pictorial representation of an example distributed dataprocessing system in which aspects of the illustrative embodiments maybe implemented;

FIG. 2 shows a block diagram of an example data processing system inwhich aspects of the illustrative embodiments may be implemented;

FIG. 3 depicts one exemplary configuration of a symmetric multiprocessor(SMP) system in the form of a processor node in accordance with anillustrative embodiment;

FIG. 4A depicts an example of one side of a processing module inaccordance with an illustrative embodiment;

FIG. 4B depicts an exemplary cubical processing module in accordancewith an illustrative embodiment;

FIG. 4C depicts an exemplary storage module in accordance with anillustrative embodiment;

FIG. 4D depicts an exemplary input/output (I/O) module in accordancewith an illustrative embodiment;

FIG. 4E depicts an exemplary filler module in accordance with anillustrative embodiment;

FIG. 5A depicts an exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment;

FIG. 5B depicts an example of a constructed processing module thatillustrates the majority of the air space within the middle of aprocessing module being filled in accordance with an illustrativeembodiment;

FIG. 6A depicts another exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment;

FIG. 6B illustrates removal of a heatsink and circuit board from aprocessing module side in accordance with an illustrative embodiment;

FIG. 6C depicts a core that may be inserted into an empty space in thecenter of a processing module in accordance with an illustrativeembodiment;

FIG. 6D illustrates another embodiment of an expanding core inaccordance with an illustrative embodiment;

FIG. 7A depicts an example of a scalable space-optimized andenergy-efficient computing system in accordance with an illustrativeembodiment;

FIG. 7B depicts an exemplary frame of a ubiquitous high-performancecomputing (UHPC) system in accordance with an illustrative embodiment;

FIG. 7C depicts an exemplary top down view of a UHPC system inaccordance with an illustrative embodiment;

FIG. 7D depicts an exemplary a view of the cooling fans of a UHPC systemin accordance with an illustrative embodiment;

FIG. 8A depicts an exemplary module that is liquid tight and liquidcooled in order to increase heat dissipation in accordance with anillustrative embodiment;

FIG. 8B depicts an exemplary cooling of multiple modules by a singleheat exchanger in accordance with an illustrative embodiment;

FIG. 8C depicts an exemplary structure where the center area of a UHPCsystem is populated by one or more modules;

FIG. 8D depicts how a three-dimensional very-large-scale integration(VLSI) global routing technique may be applied to generate a layout forthe tubes in accordance with an illustrative embodiment;

FIG. 8E depicts another exemplary structure where the heat exchangersalong the walls of a module are replaced by heat exchanger panels inaccordance with an illustrative embodiment; and

FIG. 8F depicts a heat exchanger panel in accordance with anillustrative embodiment.

DETAILED DESCRIPTION

The illustrative embodiments provide a ubiquitous high-performancecomputing (UHPC) system that packages the thousands of components of ahigh-performance computing (HPC) system into building-block modules thatmay be coupled together to form a space-optimized and energy-efficientproduct. The illustrative embodiments also provide for various heatsinkdesigns that enable an elegant assembly and in place maintenance for theheatsink and the module, while maintaining large effective heat exchangearea and high pressure for efficient cooling. The illustrativeembodiments also provide for an alternative to air cooling using aliquid cooling system with coolant/air heat exchanging enabled by skinheat exchangers mounted either on the interior or the exterior surfaceof the UHPC system.

Thus, the illustrative embodiments may be utilized in many differenttypes of data processing environments including a distributed dataprocessing environment, a single data processing device, or the like. Inorder to provide a context for the description of the specific elementsand functionality of the illustrative embodiments, FIGS. 1-3 areprovided hereafter as example environments in which aspects of theillustrative embodiments may be implemented. While the descriptionfollowing FIGS. 1-3 will focus primarily on a single data processingdevice implementation of a ubiquitous high-performance computing (UHPC)system, this is only an example and is not intended to state or implyany limitation with regard to the features of the present invention. Tothe contrary, the illustrative embodiments are intended to includedistributed data processing environments and embodiments in which aubiquitous high-performance computing (UHPC) system may easily beimplemented.

With reference now to the figures and in particular with reference toFIGS. 1-3, example diagrams of data processing environments are providedin which illustrative embodiments of the present invention may beimplemented. It should be appreciated that FIGS. 1-3 are only examplesand are not intended to assert or imply any limitation with regard tothe environments in which aspects or embodiments of the presentinvention may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

With reference now to the figures, FIG. 1 depicts a pictorialrepresentation of an example distributed data processing system in whichaspects of the illustrative embodiments may be implemented. Distributeddata processing system 100 may include a network of computers in whichaspects of the illustrative embodiments may be implemented. Thedistributed data processing system 100 contains at least one network102, which is the medium used to provide communication links betweenvarious devices and computers connected together within distributed dataprocessing system 100. The network 102 may include connections, such aswire, wireless communication links, or fiber optic cables.

In the depicted example, ubiquitous high-performance computing (UHPC)server 104 and server 106 are connected to network 102 along withstorage unit 108. In addition, clients 110, 112, and 114 are alsoconnected to network 102. These clients 110, 112, and 114 may be, forexample, personal computers, network computers, or the like. In thedepicted example, UHPC server 104 provides data, such as boot files,operating system images, and applications to the clients 110, 112, and114. Clients 110, 112, and 114 are clients to UHPC server 104 in thedepicted example. Distributed data processing system 100 may includeadditional servers, clients, and other devices not shown.

In the depicted example, distributed data processing system 100 is theInternet with network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, the distributed data processing system 100 may also beimplemented to include a number of different types of networks, such asfor example, an intranet, a local area network (LAN), a wide areanetwork (WAN), or the like. As stated above, FIG. 1 is intended as anexample, not as an architectural limitation for different embodiments ofthe present invention, and therefore, the particular elements shown inFIG. 1 should not be considered limiting with regard to the environmentsin which the illustrative embodiments of the present invention may beimplemented.

With reference now to FIG. 2, a block diagram of an example dataprocessing system is shown in which aspects of the illustrativeembodiments may be implemented. Data processing system 200 is an exampleof a computing system in which computer usable code or instructionsimplementing the processes for illustrative embodiments of the presentinvention may be located.

In data processing system 200, ubiquitous high-performance computing(UHPC) server 202 is connected to network 206 along with storage unit208 and client 204. UHPC server 202 may further comprise one or more ofcompute modules 210, storage modules 212, and input/output (I/O) modules214 using interconnect 216. Data processing system 200 may includeadditional servers, clients, storage devices, and network connects notshown. As with network 102 of FIG. 1, network 206 may represent aworldwide collection of networks and gateways that use any type ofcommunication protocols to communicate with one another. Additionally,data processing system 200 may also be implemented to include a numberof different types of networks, such as for example, an intranet, alocal area network (LAN), a wide area network (WAN), or the like. FIG. 2is intended as an example of a UHPC system, not as an architecturallimitation for different embodiments of the present invention, andtherefore, the particular elements shown in FIG. 2 should not beconsidered limiting with regard to the environments in which theillustrative embodiments of the present invention may be implemented.

FIG. 3 depicts one exemplary configuration of a symmetric multiprocessor(SMP) system in the form of processor node 300 in accordance with anillustrative embodiment. Processor node 300 may contain one or more ofservice processor 302, I/O hubs 306, fabric expansion port 308, andoff-node fabric expansion ports 310. Fabric expansion port 308 and offnode fabric expansion ports 310 provide connectivity for A and B ports312 from each of multi-chip modules (MCM) 314 to multi-chip modules onother processor nodes. Fabric ports X, Y, and Z 316 interconnectmulti-chip modules 314 within processor node 300.

Additionally, stacked memory chips 323 provide processor memory at eachMCM 314. Each of multi-chip modules 314 may be identical in its hardwareconfiguration but configured by firmware during system initialization tosupport varying system topologies and functions as, e.g. enablement ofmaster and slave functions or connectivity between various combinationsof multiple nodes in a scalable multi-node symmetric multi-processorsystem.

Within a particular multi-chip module 314 there may be found processorunit 320 that may comprise one or more processor cores. Processor node300 may have one or more oscillators 324 routed to each chip found onprocessor node 300. Connections between oscillators 324 and functionalunits extend throughout the board and chips but arc not shown in FIG. 3.Similarly, it is understood that many convoluted interconnects existbetween fabric expansion port 308, off-node fabric expansion ports 310,and I/O hubs 306 to the various chips on the board, such as A and Bports 312 and I/O ports 326 of multi-chip module 314, among othercomponents, though such interconnects are not shown in FIG. 3. Theconfiguration illustrated in FIG. 3 is just one example of a processingnode configuration as is merely shown for illustration purposes. One ofordinary skill in the art would recognize that other processing nodeimplementations may be used in the illustrative embodiments withoutdeparting from the spirit and scope of the invention.

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 1-3 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 1-3. Also, theprocesses of the illustrative embodiments may be applied to amultiprocessor data processing system, other than the SMP systemmentioned previously, without departing from the spirit and scope of thepresent invention.

Again, the illustrative embodiments provide a ubiquitoushigh-performance computing (UHPC) system that packages the thousands ofcomponents of a high-performance computing (HPC) system intobuilding-block modules that may be coupled together to form aspace-optimized and energy-efficient product. In a first embodiment, amodular processing device is composed of a plurality of identicalprinted circuit boards and processing nodes housed in identicalprocessor packages referred to as processing modules. Each processingmodule comprises memory, processing layers, and connectivity to power,other processing nodes, storage, input/output (I/O), or the like. Theconnectivity may be provided through wire, wireless communication links,or fiber optic cables and/or interconnects. In the processing module,various heatsink designs remove heat from the components on eachprocessing node. In addition to the processing module, storage and I/Omodules are also provided in similarly formed modules. The storage andI/O modules may be composed of a plurality of printed circuit boardsmounting solid state storage devices and/or optical interconnects. Thephysical design of the modules offers advantages in communicationbandwidth, cooling, and manufacturing costs.

In other embodiments, the heatsink designs arc a composite design thatis comprised of two components: the per processing node coolingcomponent and the common core component. Each heatsink component ismounted directly on one or more processing nodes. Since air flow tendsto follow the path of least resistance, one heatsink design fills amajority of the air space such that the flow of air passes between thefins of the heatsink increasing the heat exchange surface area. Inanother heatsink design, the sizing of the heatsink allows the removalof the heatsink and the processing node from the processing module,while the three other heatsinks remain in place. To enable this type ofheatsink design, an empty space is left in the center of the module.Since air flow tends to follow the path of least resistance, toeliminate the loss of beneficial air flow over the heatsinks, a core isinserted into the empty area of the module to fill the air gap,increasing the heat exchange surface area of the heatsinks. The core maybe either a solid core that air flows around, increasing the airpressure on the board mounted heatsinks, or may be another heatsink thatincreases the heat exchange surface area. Since the core is removable,it is still possible to perform in place maintenance tasks on themodule, without dissembling the module.

In another embodiment, the modules are combined to create a scalablespace optimized and energy efficient ubiquitous high performancecomputing (UHPC) system. The UHPC system reduces communication cost,reduces cooling cost, provides reliable operation, and facilitatesmaintainability. The UHPC system does so by using a modular design,where processing nodes are built as modules and assembled as a hexadron(non-regular cube) according to the computing needs of the end-user.This arrangement results in a reduced distance for the communicationlinks, which allows an all-to-all solution.

In still another embodiment, the processing, storage, and/or I/O modulesare constructed such that the modules are liquid tight and are thenliquid cooled in order to increase heat dissipation. Using liquidcooling provides for more modules to be placed in a UHPC system. Inorder to cool the liquid flowing through the modules, heat exchangersare coupled to the external surfaces of a UHPC system. Pumping themodule coolant between the modules and the heat exchangers circulatesthe module coolant through the heat exchange elements. Using theexternal surface of the UHPC system allows heat to be dissipated usingambient air.

While the following embodiments are described with relation to a moduleof cubical design, the illustrative embodiments are not limited to onlya cubical design. That is, other three-dimensional geometricconfigurations may also be used, such as a rectangular box, withoutdeparting from the spirit and scope of the present invention.

FIG. 4A depicts an example of one side of a processing module inaccordance with an illustrative embodiment. In FIG. 4A, processingmodule side comprises one or more of processing nodes 404 coupled tocircuit board 406. Each of processing nodes 404 may comprise memory,processing layers, and connectivity to other ones of processing nodes404 either coupled directly to circuit board 406 or coupled viaconnectors 408 to processing nodes on other circuit boards. Processingnodes 404 may be coupled directly to circuit board 406 in a manner inwhich if one of processing nodes 404 fail, the processing node mayremoved and replaced with a functional processing node. Similarly,circuit board 406 may also be coupled to processing module side 402 in amanner in which if circuit board 406 fails, the entire circuit board 406may removed and replaced with a functional circuit board 406. Each ofconnectors 408 may be any type of connector that provides connectivityto power, other circuit boards, storage, input/output (I/O), or thelike. The connectivity provided by connectors 408 may be wire, fiberoptic, or the like.

FIG. 4B depicts an exemplary cubical processing module 400 in accordancewith an illustrative embodiment. In FIG. 4B processing module 400partially constructed showing three processing module sides 402 a, 402b, and 402 c coupled together. As shown in FIG. 4B, processing moduleside 402 b is coupled to processing module side 402 c via connectors,such as connectors 408. Furthermore, processing module side 402 a isshown to have exterior connector 414 for interfacing with a backplane ofa ubiquitous high performance computing (UHPC) system, which will bedescribed in detail below. While only one of exterior connector 414 isshown, depending on implementation, more than one of external connector414 may be required for interfacing to the UHPC system.

FIG. 4C depicts an exemplary storage module in accordance with anillustrative embodiment. Storage module 420 may comprise storagecontroller 422 mounted to storage module side 424. Storage module 420may also comprise one or more of storage cards 426, which each comprisea plurality of storage devices 428 for storing data, such as storageclass memory chips, along with card specific controller chips 436 forinterfacing with storage controller 422. Each of storage cards 426 maybe coupled to storage card interface 430 which may be coupled to storagecontroller 422 via connector 432 on storage card interface 430 andconnector 434 on storage controller 422. Furthermore, storage moduleside 424 may have one or more exterior connectors (not shown) forinterfacing with a backplane of a ubiquitous high performance computing(UHPC) system, which will be described in detail below.

FIG. 4D depicts an exemplary input/output (I/O) module in accordancewith an illustrative embodiment. I/O module 440, which may also bereferred to as a network module, provides connectivity for the UHPCsystem to the outside world. I/O module 440 may comprise a plurality ofnetwork interface cards 442 as well as one or more adapters 444 mountedto I/O module side 446. Each of network interface cards 442 may furthercomprise a plurality of pass-thru optical connections 448, which may beused to connect multiple levels of modules in the UHPC system together.Each of network interface cards 442 may also comprise a plurality ofvery high speed Ethernet or Infiniband connectors 450.

FIG. 4E depicts an exemplary filler module in accordance with anillustrative embodiment. Filler module 460 comprises top and bottomrotating slides 462 to control the airflow thru filler module 460.Rotating slides 462 provide for different airflow impedances so thatdepending on the position the filler module is used within the UHPCsystem, rotating slides 462 may be adjusted to mimic the airflow ofprocessing module 400, storage module 420, or I/O module 440. That is,rotating slides 462 of filler module 460 may be adjusted to providedifferent airflow impedances such that the airflow impedance ofdifferent module types may be matched. This ensures that when fillermodule 460 is used, air flow properties do not change in other areas ofthe UHPC system.

FIG. 5A depicts an exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment. Asopposed to heatsinks that have common sized fins, heatsink 502 of aprocessing module is shaped such that fins 504 on the far edges ofheatsink 502 are shorter in length than a length of fin 506 in themiddle of heatsink 502. Heatsink 502 is constructed in a manner suchthat within processing module 500 with four heatsinks, the heatsinksfill a majority of the air space within the middle of processing modulesuch that the flow of air passes between the fins of the heatsinkincreasing the heat exchange surface area. As is also shown in FIG. 5A,heatsink 502 is of a width and depth such that heatsink 502 covers allprocessing nodes 508 on processing module side 510. Heatsink 502 may beheld in place over processing nodes 508 using generally knownrestrictive methods, such as retaining clips, screws, or the like.Dimension orientation 512 depicts height, width, and depth with relationto the description of heatsink 502. Heatsink 502 may be constructed fromeither copper, aluminum, or another thermally conductive material.

However, in an alternative embodiment (not shown), in order to providefaster access to processor nodes 508 during maintenance, heatsink 502may be of a width and depth to cover only one portion of processor nodes508, which would require another one of heatsink 502 to cover the otherportion of processor nodes 508 such that the majority of the air spacewithin the middle of processing module is still filled such that theflow of air passes between the fins of the heatsink increasing the heatexchange surface area. For example, one smaller depth heatsink may covertwo processor nodes while a similar smaller depth heatsink covers twoother processor nodes. While the illustrative embodiments show four ofprocessor nodes 508 on processing module side 510, the illustrativeembodiments recognize that more or fewer processing nodes may beimplemented such that the width and depth of heatsink 502 requireschanging while the height of heatsink 502 in conjunction with other oneof heatsink 502 still fill a majority of the air space within the middleof processing module such that the flow of air passes between the finsof the heatsink increasing the heat exchange surface area. FIG. 5Bdepicts an example of a constructed processing module that illustratesthe majority of the air space within the middle of processing module 500being filled in accordance with an illustrative embodiment.

FIG. 6A depicts another exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment. Again,as opposed to heatsinks that have a common sized fins, heatsink 602 ofprocessing module 600 is shaped such that a plurality of fins 604 towardthe ends of heatsink 602 are shorter in length than a plurality of fins606 in the middle of heatsink 602. Heatsink 602 is of a width and depthsuch that heatsink 602 covers all processing nodes 608 on processingmodule side 610. Heatsink 602 may be held in place over processing nodes608 using generally known restrictive methods, such as retaining clips,screws, or the like. Dimension orientation 612 depicts height, width,and depth with relation to the description of heatsink 602.

However, in an alternative embodiment (not shown), in order to providefaster access to processor nodes 608 during maintenance, heatsink 602may be of a width and depth to cover only one portion of processor nodes608, which would require another one of heatsink 602 to cover the otherportion of processor nodes 608. For example, one smaller depth heatsinkmay cover two processor nodes while a similar smaller depth heatsinkcovers two other processor nodes. While the illustrative embodimentsshow four of processor nodes 608 on processing module side 610, theillustrative embodiments recognize that more or fewer processing nodesmay be implemented such that the width and depth of heatsink 602requires changing. Heatsink 602 is constructed in a manner such thatwithin processing module 600 with four of heatsink 602, the sizing ofthe heatsinks allows the removal of the heatsink 602 and circuit board603 from the processing module side 610, while the three other heatsinksremain in place, as is shown in FIG. 6B. Circuit board 603 may becoupled to processing module side 610 by a mechanical mechanism, amagnetic mechanism, or the like, such that circuit board 603 may beeasily removed and replaced. However, to enable this design of heatsink602, an empty space is left in the center of processing module 600.

FIG. 6C depicts a core that may be inserted into an empty space in thecenter of a processing module in accordance with an illustrativeembodiment. Core 620 may be inserted into the empty area of processingmodule 600 to fill the air gap left by the use of four of heatsinks 602.Core 620 may be either a solid or impervious core that air flows aroundand across heatsinks 602 or may be another heatsink (pervious) thatincreases the heat exchange of heatsinks 602. If core 620 is animpervious core, then core 620 may be comprised of non-heat conductivematerial, such as rubber, plastic, or the like. If core 620 is of apervious to airflow design in order to provide added heat exchange toheatsinks 602, then core 620 may be comprised of a thermally conductivematerial, such as copper, aluminum, or the like.

If core 620 is to provide additional heat exchange, then core 620 may becomprised of multiple core sections 622 as shown in overhead view 624.Core sections 622 may be attached in various methods so that onceinserted in the empty area between heatsinks 602 in processing module600, core 620 may expand to maintain thermal conduction of each ofheatsinks 602. In one embodiment, core sections 622 may be coupled usingexpansion mechanisms, such that when a plurality of retention fasteners,such as latches, snap rings, pins, or the like, are released at the topand bottom of core 620, expansion mechanisms 630 expand forcing coressections 622 apart and onto heatsinks 602 as in show in expansion view626. When core 620 is to be removed, a user may use the plurality oflatches, snap rings, pins, or the like, to pull core sections 622 backtogether and away from heatsinks 602 so that core 620 may easily beremoved. In another embodiment, core sections 622 may be coupled usingretention mechanisms 632, such that when expansion rod 634 is insertedinto the center of core 620, retention mechanisms 632 are forced toexpand which forces cores sections 622 apart and onto heatsinks 602 asin show in expansion view 628. When the expansion rod is removed fromthe center of core 620 retention mechanisms 632 pull the cores sections622 back together and away from heatsinks 602 so that core 620 mayeasily be removed.

The use of core 620 allows maintenance to be performed on processingmodule 600 without dissembling processing module 600. In order toincrease heat conductivity between heatsinks 602 and core sections 622,the edges of core 620 that will come in contact with heatsink 602 may becoated with a thermal conductive paste prior to being inserted into theempty area between heatsinks 602 in processing module 600.

FIG. 6D illustrates another embodiment of an expanding core inaccordance with an illustrative embodiment. In FIG. 61), core 640comprises four core sections 642 with a center pin 644. Center pin 644,which may be removable, at the center of core 640 acts as a guide toprevent core 640 from touching the board mounted heatsinks. While notshown, center pin 644 may be affixed to a aerated base plate at thebottom of module 600 or a stack of modules such that when core 640 isinserted into the module or stack of modules, center pin 644 acts as aguide so that the exterior edges of core 640 does not contact anyheatsink in the module. This is especially important to maintain thethermal material on the surfaces of core 640 that will come in contactwith the board mounted heatsinks, preventing the thermal material frombeing wiped out during sliding core 640 into the module. The thermalmaterial and the applied pressure between core 640 and the heatsinks ofthe processing module are both important to maintain high thermalconductivity between the two components for efficient cooling of core640. When wedge 646 is inserted into core 640, core 640 expands so thatthe sides of core 640 come into contact with the heatsinks of theprocessing module as is shown in expansion views 648 and 650.

FIG. 7A depicts an example of a scalable space-optimized andenergy-efficient computing system in accordance with an illustrativeembodiment. In FIG. 7A, a ubiquitous high performance computing (UHPC)system 700 provides a compact arrangement of modules 702 configured inframe 704 that reduces communication cost, reduces cooling cost,provides reliable operation, and facilitates maintainability. Themodular design of UHPC system 700 provides these benefits by assemblingthe modules in a hexadron (non-regular cube) according to the computingneeds of the end-user which reduces distance for the communicationlinks. UHPC system 700 comprises frame 704, modules 702, air inlet 706,air mixing plenum 708, and one or more cooling fans 710. Each of modules702 may be either a processing module, a storage module, an input/output(I/O) module, or a filler module, as previously described, and may beinstalled in frame 704 similar to a drawer as is illustrated. The othercomponents of UHPC system 700 will now be described in detail.

FIG. 7B depicts an exemplary frame of a UHPC system in accordance withan illustrative embodiment. Frame 704 provides a plurality of identicalmodule compartments 712 such that any type of module may be insertedinto a single one of module compartment 712 and be connected viabackplane 714 to power, storage, communication, or whatever access isrequired by the module. Frame 704 provides sections between each ofmodule compartments 712 so that cabling may be run between the variousconnectors of the backplanes as well as to external power and networkconnects for environments where UHPC system 700 is deployed.Additionally, the top and bottom of each of module compartments 712 areopen so that air may flow through each column of module compartments 712from the air inlet 706 to air mixing plenum 708. Each level of modulecompartments may also be individual sections such that UHPC system 700may comprise as few as one level up to any number of levels such thatthe power and cooling needs are still met by UHPC system 700.

Air inlet 706 may be a compartment that has a solid bottom with opensides and top. Each of the sides of air inlet 706 may be constructedsuch that access panels provide for the insertion and replacement of airfilters. Air would flow through the air filters in the sides of airinlet 706 and up through the top of air inlet 706 through modulecompartments 712 to air mixing plenum 708. The top of air inlet 706 maybe constructed in a way that the top section provides knock outs in thedimensions of module compartments 712 so that a user may remove onlythose knock outs for those columns of module compartments 712 that arepopulated in frame 704. Using knock outs in air inlet 706 allows theuser to cool only those areas of frame 704 that are occupied by modules702. Further, in the event a knock out is erroneously removed or ifmodules 702 are removed such that a column of module compartments 712 nolonger has any occupying modules 702, filler plates may be provided toreplace the knock out.

FIG. 7C depicts an exemplary top down view of a UHPC system inaccordance with an illustrative embodiment. As can be seen in FIG. 7C,air mixing plenum 708 may be a sectional compartment that is placedabove the top level of module compartments 712 and covers the outsideperimeter of frame 704 such that air flowing up through modulecompartments 712 will be accumulated in the area of air mixing plenum708. Also show in FIG. 7C is center area 716 which is shown as empty butmay be used for cabling, other modules, or the like. The use of centerarea 716 for modules is illustrated in a different embodiment that isdescribed below. In this illustration, center area 716 may not have airflow as restricted by air inlet 706 previously described.

FIG. 7D depicts an exemplary view of the cooling fans of a UHPC systemin accordance with an illustrative embodiment. In FIG. 7D there areshown four cooling fans 710 each of which draw air though the air inletat the bottom of the UHPC system, through one or more modulecompartments, and through the air mixing plenum. Each of fans 710 may becontrolled either individually or as a group. That is, dependent onsensed temperature in the UHPC system, fans 710 may be controlled suchthat one fan turns on when the temperature exceeds one thresholdtemperature and the other fans may individually turn on as othertemperature associated thresholds are exceeded. Likewise, the fans mayindividually turn off as temperature levels within the UHPC systemdecrease and the associated temperature thresholds are no longerexceeded. The temperature thresholds may be controlled through simplethermostats associated with each fan or other more complex thermalcontrols. In an alternative embodiment, fans 710 may be a single fanthat has a variable motor that increases to draw more air as thetemperature of the UHPC system increases.

Additionally, while the exemplary embodiment illustrates four of fans710, the illustrative embodiment recognizes that more or fewer fans maybe used without departing from the spirit and scope of the invention.Further, while fans 710 are shown on top of the UHPC system, fans 710may be placed anywhere in the UHPC system such that air is pushed orpulled through the UHPC system. For example, fans 710 may be locatedbelow the air inlet, or between the air inlet and the modulecompartments.

FIG. 8A depicts an exemplary module that is liquid tight and liquidcooled in order to increase heat dissipation in accordance with anillustrative embodiment. In FIG. 8A, module 800 is constructed similarto the modules previously described but also has a top and bottom sidethat causes the module to be liquid tight other than inlet port 802 andoutlet port 804, as well as electrical and optical connections. Inletport 802 and outlet port 804 may be located at opposing locations onmodule 800, such that module 800 may be filled with a non-conductiveliquid and pump 812 located within or near the module may pump theliquid through module 800. As devices within the module heat thenon-conductive liquid, the hot liquid flows out of or exits module 800via outlet port 804 into exit tube 808 and through heat exchanger 806which may be located on and coupled to an outside air exposed area ofthe UHPC system, preferably the module with which the heat exchanger isassociated. Exposure of the liquid to the ambient air around the UHPCsystem through heat exchanger 806 cools the liquids such that, after theliquid finishes its pass through heat exchanger 806, the liquid isreturned through return tube 810 back into module 800 via inlet port 802at a cooler temperature than when it exited module 800. While pump 812is shown to be located in line with exit tube 808, pump 812 may belocated either in the exit line or the return line, whichever isdetermined to be more efficient. The pump may also be located in acavity within the heat exchanger, dedicated to host the pump, in orderto make it easily accessible for repair and routine pump maintenance.

While not shown in FIG. 8A, the components within module 8A may becooled with the use of a cold plate mounted on each circuit board, suchas circuit board 603 in FIG. 6B, replacing a heatsink, such as heatsink602 of FIG. 6B. Coolant may be circulated through each cold plate from acommon inlet port 802 and out a common outlet port 804. In thisembodiment the cooling fluid may be conducting and filling the modulewith cooling fluid is not required. Thus, module 800 may not need to beliquid tight. In this embodiment only the interior cooling components ofthe module are different than the other embodiments discussed.

FIG. 8B depicts an exemplary cooling of multiple modules by a singleheat exchanger in accordance with an illustrative embodiment. In thisembodiment, heat exchanger 806 provides sufficient cooling such that theliquid may be passed through more than one of modules 800 before beingcooled again in heat exchanger 806. In this embodiment, two or more ofmodules 800 are configured such that outlet port 804 of a first module800 is coupled to the inlet of its heat exchanger 806 via exit tube 808and the outlet of heat exchanger 806 is coupled to inlet port 802 of asecond module 800 via return tube 810. Then outlet port 804 of thesecond module 800 is coupled to inlet port 802 of the first module 800using coupling tube 814.

FIG. 8C depicts an exemplary structure where the center area of a UHPCsystem is populated by one or more modules. In this illustrativeembodiment, liquid tight center modules 820 may be inserted within agiven module compartment and coupled to a backplane in order to obtainconnectivity to power, communications, storage, or the like. Since thesecenter modules 820 do not have direct access to an outside air exposedarea of the UHPC system, extension tubes 815 are used to give the centermodules 820 access to the heat exchangers 806 on the exterior perimeterof the UHPC system. In this illustrative embodiment, concern is givenwith regard to the length of the exit tube and the return tube (showncombined as extension tubes 815) associated with each of modules 800 andcenter modules 820 so that the shortest distance to and from the heatexchanger is provided.

FIG. 8D depicts how a three-dimensional very-large-scale integration(VLSI) global routing technique may be applied to generate a layout forthe tubes, such as exit tubes 808, return tubes 810, coupling tube 814,and extension tube 815 of FIGS. 8A-8C, in accordance with anillustrative embodiment. To solve the problem of routing the heatexchangers to the modules, simple graph 820 is constructed thatrepresents the topology of the modules. There is one vertex 822 permodule and a single undirected edge 821 that connects the modules, whichabut each other. Edges 821 represent the channels through which tubesare allowed to pass. Each edge 821 is given a capacity which representsthe number of tubes which are allowed to pass between adjacent modules.In conventional VLSI global routing, connections of the components areknown as a priori, hence fixed. For the routing of the tubes in a UHPCsystem, there is only one known endpoint per connection and a heatexchanger must connect to each module which requires cooling. To adaptthe VLSI global routing techniques to this domain, a unique air vertex823 is added, which represents the air where heat is eventuallyradiated. From air vertex 823, undirected edges 821 are added to all ofthe possible locations where the heat exchangers may be attached. Foreach of edges 821 that connect air vertex 823 to the 3D mesh, a capacityis assigned that is equal to the number of modules with which the heatexchanger is allowed to be connected. For the remainder of edges 821,their capacities are set as the number of tubes, such as exit tubes 808,return tubes 810, coupling tubes 814, and extension tube 815 of FIG.8A-8C, that are allowed to pass between the adjacent modules inquestion. Each module 800 which needs to be cooled may be connected withthe air vertex 823. A solution to this routing problem routes tubes tothe heat exchangers 806.

FIG. 8E depicts another exemplary structure where the heat exchangersalong the walls of a module are replaced by heat exchanger panels inaccordance with an illustrative embodiment. In FIG. 8E, heat exchangerpanels 832 and 833, which may be pervious or impervious, on the outsideof UHPC system 830 represent panels of heat exchangers that may be usedto cool the non-conductive liquid coolant that circulates through aplurality of modules 800. Exit tube 808 and return tube 810 transfer thenon-conductive liquid coolant to the heat exchanger panels 832 and 833.Extension tubes 815 transfer the non-conductive liquid coolant throughmodules 800. A particular UHPC system 830 may have heat exchanger panel832 and/or 833 coupled to a single module 800, in which case there is noneed to use extension tubes 815. Each of heat exchanger panels 832 and833 may be coupled to the surface of UHPC system 830 such that heatexchanger panels 832 and 833 are adjacent to the sides of UHPC system830, as is shown by heat exchanger panel 832, or each of heat exchangerpanels 832 and 833 may be coupled to the surface of UHPC system 830 suchthat heat exchanger panels 832 and 833 may be tilted away from the sidesof UHPC system 830 so that air flow may be increased across the heatexchangers, as is shown by heat exchanger panel 833. Furthermore,cooling of the non-conductive liquid coolant may be facilitated byassisting the flow of air around heat exchanger panels 832 and 833. Forthat purpose, heat exchanger panels 832 and 833 may be pervious orimpervious to air flow.

FIG. 8F depicts a heat exchanger panel in accordance with anillustrative embodiment. In FIG. 8F, heat exchanger 806 is installed inthe heat exchanger panel 832. Exit tube 808 and return tube 810 are usedto allow the non-conductive liquid coolant to cool. The area of the heatexchanger panel 832 may be smaller, equal to, or larger than the area ofthe exterior of the UHPC system 830. Heat exchanger panel 832 mightdissipate heat on one or many of its sides. For example, heat exchanger832 that is adjacent to UHPC system 830 may be designed to onlydissipate heat by the outer surface. While heat exchanger 833 that istilted with respect to UHPC system 830 might dissipate heat by the innerand the outer surfaces.

Thus, the illustrative embodiments provide a ubiquitous high-performancecomputing (UHPC) system that packages the thousands of components of ahigh-performance computing (HPC) system into building-block modules thatmay be coupled together to form a space-optimized and energy-efficientproduct. The illustrative embodiments also provide for various heatsinkdesigns that enable an elegant assembly and in place maintenance for theheatsink and the module, while maintaining large effective heat exchangearea and high pressure for efficient cooling. The illustrativeembodiments also provide for an alternative to air cooling using aliquid cooling system with coolant/air heat exchanging enabled by skinheat exchangers mounted either on the interior or the exterior surfaceof the UHPC system.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A liquid-tight module, comprising: a processingmodule top; a processing module bottom; a set of processing modulesides, wherein each processing module side comprises: a circuit board; aplurality of connectors coupled to the circuit board; and a plurality ofprocessing nodes coupled to the circuit board, wherein each processingmodule side in the set of processing module sides couples to anotherprocessing module side using at least one connector in the plurality ofconnectors such that when all of the set of processing module sides arecoupled together with the processing module top and the processingmodule bottom, the liquid-tight module is formed, and wherein theliquid-tight module is filled with a non-conductive liquid coolant usedto cool the plurality of processing nodes within the liquid-tightmodule, wherein the non-conductive liquid coolant is circulated in aclosed loop by at least one pump through a plurality of tubes and atleast one heat exchanger in a plurality of heat exchangers, wherein theat least one heat exchanger is coupled to an exterior portion of theliquid-tight module, and wherein the at least one heat exchanger coolsthe non-conductive liquid coolant using air surrounding the liquid-tightmodule.
 2. The liquid-tight module of claim 1, wherein the liquid-tightmodule shares the non-conductive liquid coolant, the plurality of tubes,and the at least one heat exchanger with multiple liquid-tight modulesin a computing system.
 3. The liquid-tight module of claim 1, whereinthe at least one heat exchanger is mounted on at least one exposedsurface of the liquid-tight module.
 4. The liquid-tight module of claim2, wherein the plurality of tubes are coupled to the multipleliquid-tight modules such that: a first end of a first tube is coupledto a first surface of a first liquid-tight module, a second end of thefirst tube is coupled to a first surface of a second liquid-tightmodule, a first end of a second tube is coupled to a second surface ofthe second liquid-tight module, wherein the first surface and the secondsurface of the second liquid-tight module are opposite each other, and asecond end of the second tube is coupled to a second surface of thefirst liquid-tight module, wherein the first surface and the secondsurface of the first liquid-tight module are opposite each other, andwherein the pump circulates the non-conductive liquid coolant inopposite directions through the multiple liquid-tight modules.
 5. Theliquid-tight module of claim 1, wherein the plurality of heat exchangersare partitioned equally between a plurality of liquid-tight modules suchthat there is one heat exchanger per liquid-tight module.
 6. Theliquid-tight module of claim 1, wherein the plurality of heat exchangersare cooled by natural air convection.
 7. The liquid-tight module ofclaim 1, wherein the heat exchanger is tilted at an oblique angleagainst a side surface of the liquid-tight module.
 8. The liquid-tightmodule of claim 1, wherein the plurality of tubes are coupled to theliquid-tight module such that: a first end of a first tube is coupled toa first surface of a first liquid-tight module, and a second end of thefirst tube is coupled to a second surface of the first liquid-tightmodule, wherein the first surface and the second surface of the firstliquid-tight module are opposite each other.